Targeted cross-validation

In many applications, we have access to the complete dataset but are only interested in prediction of a particular region predictor variables. A standard approach is find globally best modeling method from set candidate methods. However, it perhaps rare reality that one uniformly better than others. natural for this scenario apply weighted L2 loss performance assessment reflect region-specific interest. We propose targeted cross-validation (TCV) select models or procedures based on general loss. show TCV consistent selecting performing under Experimental studies used demonstrate use and its potential advantage over global CV using local data region. Previous investigations relied condition when sample size large enough, ranking two candidates stays same. applications with setup changing data-generating processes highly adaptive methods, relative methods not static as varies. Even fixed process, possible switches infinitely times. work, broaden concept selection consistency by allowing switch varies, then establish TCV. This flexible framework can be applied high-dimensional complex machine learning scenarios where performances dynamic.